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ANALYSIS OF RESPONSES TO THE STUDENT OPINION FORM 
Department of Administrative Sciences 1983-1990 



Abstract 



Student Opinion Form data on 1,610 classes taught in the 
Department of Administrative Sciences between 1983 and 1990 
were subjected to SAS analyses employing principally the 
General Linear Model procedure. The independent variables in 
the analyses were, for each class. Faculty (identified by the 
two-letter mail code). Course (e.g., MN 3111), Majority 
Curriculum (e.g., 847), Class Size, Mean Item 13 (student 
opinion of course, measured on a 5-point scale), and Proportion 
for Whom Required (the proportion of students in class required 
to take the course). The dependent variable was Mean Item 12 
(student opinion of instructor, measured on a 5-point scale). 

The principal results were that Course and Faculty 
accounted for 66% of the variation in Mean Item 12, with Course 
accounting for 35% and Faculty for the remaining 31%. 

Different from previous studies of the SOF, neither Class Size 
nor Majority Curriculum accounted for a notable percentage in 
the variation of Mean Item 12. The two together, in fact, 
accounted for considerably less than 1%. 

A major implication of these results is that the 
appropriate bench mark for comparison in evaluating an 
instructor's effectiveness in class is not the overall average 
(across all classes) of 4.03 on Mean Item 12 but the average 
for the course taught. The average for a course might vary 
considerably from the overall average. This report presents 
the Mean Item 12 averages for all 131 Administrative Sciences 
courses represented in the data set. 



ANALYSIS OF RESPONSES TO THE STUDENT OPINION FORM 



Department of Administrative Sciences 1983-1990 

Summary 

Student Opinion Form data on 1,610 classes taught in the 
Department of Administrative Sciences between 1983 and 1990 
were subjected to SAS analyses employing principally the 
General Linear Model procedure. The independent variables in 
the analyses were, for each class. Faculty (identified by the 
two-letter mail code). Course (e.g., MN 3111), Majority 
Curriculum (e.g., 847), Class Size, Mean Item 13 (student 
opinion of course, measured on a 5-point scale), and Proportion 
for Whom Required (the proportion of students in class required 
to take the course). The dependent variable was Mean Item 12 
(student opinion of instructor, measured on a 5-point scale). 

Results were that Course and Mean Item 13, along with 
Faculty, accounted for most of the variation in Mean Item 12. 
Incrementally, Mean Item 13 accounted for 79%, Course for 6%, 
and Faculty for 5% of this variation. Course and Faculty alone 
accounted for 66%, with Course accounting for 35% and Faculty 
for the remaining 31%. Different from previous studies of the 
SOF, neither Class Size nor Majority Curriculum accounted for a 
notable percentage in the variation of Mean Item 12. The two 
together, in fact, accounted for considerably less than 1%. 

Implications of these results are at least twofold. The 
first is that class size, particularly near the mean of 20.6 
students for these data, need not be a consideration in school 
instructional policy. The second is that the appropriate bench 
mark for comparison in evaluating an instructor's effectiveness 
in class is not the overall average (across all classes) of 
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4.03 on Mean Item 12 but the average for the course taught. 

The average for a course might vary considerably from the 
overall average. This report presents the Mean Item 12 
averages for all 131 Administrative Sciences courses 
represented in the data set. 

Since studies of SOF data for other departments are likely 
to lead to comparable results, particularly different Mean Item 
12 averages for different courses, the school should authorize 
these studies to provide the course or other bench marks needed 
for appropriate evaluation of instructional effectiveness 
throughout the school . 
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ANALYSIS OF RESPONSES TO THE STUDENT OPINION FORM 
Department of Administrative Sciences 1983-1990 

Background 

To facilitate instruction, the Naval Postgraduate School 
has since 1977 administered to its students a Student Opinion 
Form (SOF) developed from a form created by Educational Testing 
Service and used earlier at the school. The original purpose 
of these forms was to provide feedback to instructors. 
Subsequently the purpose has expanded to include evaluation of 
instruction for pay, promotion, and tenure decisions. 

The SOF contains 16 items that require responses on a 
five-point scale, with 1 the lowest value and 5 the highest 
value. Items 1 to 11 refer to specific aspects of instruction. 
Item 12 requests overall evaluation of the instructor, and Item 
13 requests overall evaluation of the course. The remaining 
items refer to the textbooks and other possible aspects of the 
course. Exhibit 1 shows these 16 items as they appear on the 
form. Item 12--particularly the mean for each class, called 
Mean Item 12 in this report--receives most attention from 
instructors and administrators. 

In addition to these items, the SOF requests information 
about the student's curriculvim and, for each student, the 
elective status of the course (required or not). Summary 
information for each class indicates the number of students in 
the class who completed the form. 

At the end of each academic term, the school separates the 
response data from the instructor and course information and 
stores the response data in computer-accessible form at the 
school's computer center. Each department head receives the 
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instructor and course information on a SOF-transmittal sheet 
that also contains summary information describing the response 
data for each class. The only link between the response data 
stored at the computer center and the instructor and course 
information on the SOF-transmittal sheet is the Student Opinion 
Form number (SOF number) assigned to the class. These numbers 
vary from instructor to instructor and from course to course 
over successive school terms. Their range varies from 
department to department. For the Department of Administrative 
Sciences, the SOF numbers range from 140 to 219. 

Different departments at the school have from time to time 
conducted studies of the SOF data. Professor Robert R. Read 
(1979) published a study of SOF responses by students in 
probability and statistics classes. This study showed that 
course and curriculum accounted for more variation in Mean Item 
12 (overall evaluation of instructor) than faculty did. A 
contemporaneous study by Joel Weston Aiken (1979), one of 
Professor Read's students, showed within the Operations 
Research Department that class size also affected Mean Item 12 
responses, larger classes being accompanied by lower response 
values. In a thesis study involving 28 Administrative Sciences 
classes (for which complete data were available) and extending 
over two academic terms, Vivian G. Melidosian and Carol A. 

White (1984) partly confirmed these earlier findings, 
particularly with respect to the substantial effects on Mean 
Item 12 responses of curriculum and class size, as well as 
faculty. The Melidosian and White study, which also indicated 
that the number of years an instructor has been on the NPS 
faculty had a notable effect on Mean Item 12 responses, 
included a large bibiliography on student evaluation of 
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nstruction. 



The study reported here is an extension of these earlier 
studies. The data represent 1,610 classes offered by the 
Department of Administrative Sciences from Winter Quarter of 
1983 to Summer Quarter of 1990, with Summer Quarter of 1984 
missing because of loss of the SOF-transmittal-sheet 
information for that quarter. Incomplete data also resulted in 
the omission of 46 from the originally 1,656 classes of this 
eight-year data set. 

Construction of the Data Set 

The SOF-transmittal-sheet information--the SOF summary 
information provided each department head following every 
quarter--consists of one line per class. This line contains 
the name of the instructor, the name of the course, and the SOF 
number, among other information. The response data stored at 
the computer center consists of one line per student. This 
line contains the student's numerical responses to Items 1 to 
16, the student's curriculum number, the elective status of the 
course for the student, the quarter and year of the class, and 
the class's SOF number, among other information. 

Construction of the data set began with the summarization 
of the response data for each class. A FORTRAN program, 
presented in Appendix A, carried out this summarization. The 
result was a line for each class that contained the SOF number, 
an identification of each of the first eleven items showing its 
rank (to be explained later). Mean Over 11 (the mean student 
response, on a 5-point scale, to the first eleven items over 
all the students in class). Mean Item 12 (the class mean of 
student responses to Item 12, overall evalution of the 
instructor, measured on a 5-point scale). Standard Deviation of 
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Item 12 (the standard deviation of the Item 12 responses within 
the class), Median Item 12 (the median of the class's Item 12 
responses), Mean Item 13 (the class mean of student responses 
to Item 13, overall evaluation of the course, measured on a 5- 
point scale). Majority Curriculum (the curriculum number for 
the majority of the students in class, e.g., 847), Proportion 
for Whom Required (the proportion of students for whom the 
course was required), and the Quarter (e.g., SUM for summer 
quarter) and Year (e.g., 83 for 1983) of the class. Eleven 
columns contained the ran)c information for the first eleven 
items. The item "number" of the item that had the highest mean 
rating in the class appeared in the first of these columns; 
this was a numeral from 1 to 9 for one of the first nine items, 
0 for Item 10, and * for Item 11. The "number" of the item 
that had the second highest mean rating appeared in the second 
of the eleven columns, and so on to the last of these columns, 
which contained the "number" of the item that had the lowest 
mean rating in the class. The sequence of "numbers" in these 
eleven columns represents a sort of instructional "fingerprint" 
for an instructor. 

As a result of the FORTRAN processing of the raw response 
data, a separate file now contained the response summaries for 
all the classes taught in each quarter and year, one file for 
each successive quarter in the eight-year period. The next 
task was to construct a corresponding file from the SOF- 
transmittal-sheet information. This file would contain for 
each class taught in the quarter the SOF number and the faculty 
and course designations: Faculty, identified by the two-digit 

mail code (e.g., WZ), and Course (e.g., MN 3111). 

Administrative Sciences Department staff constructed these 
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files. The final task was to join them with the response- 
summary files for each class. Using the SOF number as a link, 
a SAS program, presented in Appendix B, performed this task. 

The result was a single file of 1,610 lines containing SOF 
information for most of the classes taught in the 
Administrative Science Department between 1983 and 1990. 

Exhibit 2 shows a number of lines of this file, with faculty 
identification omitted to assure anonymity. (In this exhibit, 
IDSOF refers to SOF number, IDPROF to Faculty, IDCOURSE to 
Course, NSTUDENT to Class Size, CURR to Majority Curriculum, 
MEANll to Mean Over 11, MEAN12 to Mean Item 12, STD12 to 
Standard Deviation of Item 12, MED12 to Median Item 12, MEAN13 
to Mean Item 13, REQD to Proportion for Whom Required, and 
QTRYR to the quarter and the calendar year of the class.) 

Analysis of the Data 

The data represented not only 1,610 classes but also 151 
faculty members, 131 courses, and 23 curricula. Exhibit 3 
describes the quantitative variables, exclusive of year. 

Notable among these is Class Size (more precisely, the number 
of students who completed the SOF in each class). The mean 
Class Size was 20.6, and the standard deviation was 8.76. 
Exhibit 4 shows the intercorrelations among the quantitative 
variables, including year. 

Exhibit 5 shows the frequency distribution of Mean Item 12 
for all 1,610 classes in the data set. The distribution is 
skewed substantially to the left. Comparison of the class 
averages of Mean Item 12 and Median Item 12 in Exhibit 3--4.03 
and 4.10, respectively — indicates a similar skewness of the 
distributions of Mean Item 12 within classes. 

A. Analysis of Quantitative Variables 
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The first analysis focused on the quantitative variables 
only. The procedure used was the SAS PROC REG with Mean Item 
12 the dependent variable and Mean Item 13, Class Size, Year, 
and Proportion for Whom Required the independent variables. 
Exhibit 6 shows the results. Whereas Mean Item 13 accounted 
for 79% of the variation in Mean Item 12, the other three 
variables together accounted for considerably less than 1%. 
Statistically, however, all independent variables were 
signf leant with p < .002 except for Class Size, which was 
significant with p < .02. Mean Item 12 decreased by .01 for 
each successive year, increased (a surprise) by .01 for each 
10% increase in the number of students required to take the 
class, and increased (another surprise) by .01 for each 
increase of five students in class size. The effects of 
Proportion for Whom Required and Class Size are surprising not 
only because of their direction but also because of their 
magnitude. In contrast to these relatively modest effects, an 
increase in Mean Item 13 of 1.00 resulted in an increase of .95 
in Mean Item 12. 

The apparent directional anamolies for Class Size and 
Proportion for Whom Required are due to the presence of Mean 
Item 13 as a "dominant" variable in the regression. The 
following formula, for standardized variables, shows the effect 
on a slope coefficient for one independent variable (indexed by 
1 ) of adding another independent variable ( indexed by 2 ) to a 
regression: 

Pi = ^D1 ■ P2^12 

where D refers to Mean Item 12 (the dependent variable). 
Consider in this formula that rjj]^ is the slope coefficient for 
Class Size (variable 1) before the addition of Mean Item 13 
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(variable 2) and that is the slope coefficient for Class 
Size afterwards, when p >2 is the slope coefficient of Mean Item 
13. In this case, even though rjj]^ is actually negative (-.16, 
as indicated in Exhibit 4), can turn out to be positive (in 

fact, .002, as Exhibit 6 shows) because ^2 positive (.95, as 
Exhibit 6 also shows) and r]_2 is negative (-.23, as indicated 
in Exhibit 4). The same reasoning applies to Proportion for 
Whom Required with its simple correlation of -.24 with Mean 
Item 12 (as shown in Exhibit 4) despite its slope coefficient 
of .001 in the multiple regression involving Mean Item 13 (see 
Exhibit 6). In this multiple regression. Mean Item 13 carries 
the negative effects of Proportion for Whom Required and Class 
Size because of its own negative correlation with each of these 
two variables and its dominating positive correlation with Mean 
Item 12. 

B. Analysis of Categorical Variables 

Because the data involved a mixture of quantitative and 
categorical variables that were intercorrelated, the analytical 
tool for the remaining analyses was the SAS GLM (General Linear 
Model) procedure. Along with perhaps other independent 
variables, quantitative or categorical, this procedure enters a 
categorical variable into a regression analysis as a sequence 
of dummy (0-1) variables equal in number to one less than the 
number of categories. The results show the portions of total 
variation due to error (residual sum of squares), due to the 
model (regression sum of squares), due incrementally to each 
independent variable as it enters the analysis (Type I Sum of 
Squares), and due to the residual part of each independent 
variable in its regression on the other independent variables 
in the analysis (Type III Sum of Squares). Included optionally 
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among the results are dependent-variable means for all values 
or for all combinations of values of the categorical variables. 
In all GLM analyses conducted, the dependent variable was Mean 
Item 12. 

The first GLM analysis involved Faculty (e.g., WZ), Course 
(e.g., MN 3111), and Majority Curriculum (e.g., 847) as the 
independent variables. This analysis showed that for these 
data, different from the data analyzed in previous studies. 
Majority Curriculum did not account for a statistically 
significant portion of the variation in Mean Item 12 (p > .05). 
Faculty and Course did, however, account for a significant 
portion of the total variation (p < .0001). 

The second GLM analysis involved only Faculty and Course 
as independent variables. This analysis entered Course first 
in the regression. Exhibits 7 and 8 show the results. In the 
pie chart (Exhibit 8), which is based on the table in Exhibit 
7, the Faculty portion represents the variation in Mean Item 12 
due to Faculty alone, the Course portion represents the 
variation in Mean Item 12 due to Course alone, and the "Common" 
(white) portion represents the variation due in common to both. 
(This common variation reflects the interrelationship between 
Faculty and Course: A faculty member tends to teach some 

courses, but not others.) Exhibit 7 reveals that, together. 
Faculty and Course accounted for 66% of the variation in Mean 
Item 12 (r 2 = .66) and that, separately (see the Type III SS 
column). Faculty accounted for 31% (100 X 254/825) and Course 
for 8% (100 X 63/825). As the Type I SS column in Exhibit 7 
shows, however. Course prior to the entry of Faculty accounted 
for 35% of the variation in Mean Item 12 (100 X 288/825) while 
Faculty incrementally accounted for only 31% (100 X 254/825). 
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These two percentages, 35% and 31%, add up to the total 
percentage (66%) of the variation in Mean Item 12 accounted for 
by both Faculty and Course. 

C. Analysis of Quantitative and Categorical Variables 

The importance of the 35% variation due to Course is that 
Course is not student opinion of course, which is Mean Item 13. 
Course and Mean Item 13, however, have a moderately strong 
correlation (or "correlation ratio," since one of the variables 
is categorical): .63. Student opinion of a course does indeed 

vary with the course. The third GLM analysis involved Mean 
Item 13, entered first, along with Course and Faculty as 
independent variables. The three variables together accounted 
for 90% of the variation in Mean Item 12 (R^ = .90). Exhibits 
9 and 10 show the results. Whereas separately, as shown in 
Exhibit 10(a), Mean Item 13 accounted for 24%, Course for 2%, 
and Faculty for 5%, incrementally (starting from zero), as 
shown in Exhibit 10(b), Mean Item 13 accounted for 79%, Course 
for 6%, and Faculty for 5%. The three incremental percentages 
--79%, 6%, and 5%--add up to the total percentage (90%) of the 
variation in Mean Item 12 accounted for by Mean Item 13, 

Course, and Faculty together. 

Implications 

These results suggest that the instructor alone (Faculty), 
independently of the course (Course) and the student opinion of 
the course (Mean Item 13), may account for little variation 
(5%) in the overall evaluation of the instructor (Mean Item 
12). Different from Mean Item 13, which li)cely has a causally 
reciprocal relationship with Mean Item 12, Course is arguably 
an "exogenous" variable, having only a unidirectional causal 
effect on Mean Item 12. The practical implication of these 
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results is that an instructor's Mean Item 12 for a class be 
measured, not against the average Mean Item 12 for all classes, 
but against the average Mean Item 12 for the course taught. 

The average Mean Item 12 for all 1,610 classes in the data 
set is 4.03. Measured against this standard, an instructor 
with a Mean Item 12 of 3.65 might not fare so well. What if 
the course taught were MN 3172 which, for 44 classes, has an 
average Mean Item 12 of 3.54? Measured against this (the 
appropriate) standard, the instructor is seen to have performed 
notably above average. Appendix 3 shows the average Mean Item 
12 values for the 131 courses taught in the Department of 
Administrative Sciences between 1983 and 1990. 

The results reported here have at least one other 
implication. Class size does not seem to affect the overall 
evaluation of an instructor to any notable extent. The 
implication of this finding seems to be that the school need 
not seel< to reduce class size in efforts to improve instruction 
or, at least, reported student perception of instructional 
quality. This implication, of course, applies to the range of 
class sizes in the data set (from 2 to 86, with a mean of 20.6 
students). For a different range, particularly for a range 
with a higher mean, the finding might be quite different. 

SOF studies in departments other than Administrative 
Sciences, liJcely to lead to comparable results, may be 
necessary to assure appropriate evaluation of instructional 
effectiveness throughout the school. 
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35 


365 


4230819X765 


4.06 


4.06 


0.75 


4.07 


3.77 


0.97 


FAL83 


6 


155 




CM4925 


0 


18 


620 


8324X109756 


4.50 


4.50 


0.69 


4.68 


4.11 


0.72 


FAL83 


7 


156 




IS2000 


0 


46 


367 


89X32104675 


3.31 


3.04 


1 . 18 


3.03 


3.02 


0.98 


FAL83 


8 


157 




IS3171 


0 


32 


367 


3409281X675 


3.82 


3.91 


0.88 


3.96 


3.19 


0.78 


FAL83 


9 


158 




IS3183 


1 


31 


837 


09284X13756 


4.13 


4.00 


0.88 


4.07 


3.16 


0.97 


FAL83 


10 


159 




IS3183 


2 


30 


847 


0924381X576 


4.17 


4.13 


0.62 


4.11 


3.30 


0.90 


FAL83 


11 


160 




IS3186 


1 


2 


847 


X0987654321 


5.00 


5.00 


0 . 00 


5.00 


5.00 


0 . 00 


FAL83 


12 


161 




IS4183 


0 


17 


367 


4098251X736 


3.76 


3.76 


0.88 


3.75 


3.29 


0.47 


FAL83 


13 


162 




IS4185 


1 


19 


367 


41238X90756 


4.38 


4.47 


0.99 


4.77 


4.21 


0.56 


FAL83 


1^ 


163 




IS4185 


2 


17 


367 


41823X09765 


4.57 


4.71 


0.75 


4.89 


4.18 


0.53 


FAL83 


15 


164 




IS4200 


1 


23 


368 


093X4281576 


3.90 


3.57 


0.71 


3.55 


3.22 


0.86 


FAL83 


16 


165 




IS4200 


2 


19 


368 


920438X1567 


4.21 


4.11 


0.72 


4.11 


3.72 


0.89 


FAL83 


17 


166 




Mrasoi 


0 


9 


360 


X2031846975 


4.28 


4.44 


0.68 


4.60 


3.89 


0.29 


FAL83 


18 


167 




MN2106 


0 


7 


620 


841032X9765 


4.18 


4.29 


0.45 


4.20 


3.43 


0.86 


FAL83 


19 


168 




MN2155 


1 


31 


367 


304X8629175 


4.00 


3.94 


0.84 


3.96 


3.55 


0.93 


FAL83 


20 


169 




MN2155 


2 


27 


367 


04389x62175 


4.40 


4.52 


0.57 


4.60 


4.04 


0.96 


FAL83 


21 


170 




MM3001 


0 


11 


857 


32471098X65 


3.78 


3.36 


1.07 


3.60 


3.55 


0.90 


FAL83 


22 


171 




MN3105 


1 


22 


367 


8921043X756 


4.12 


3.95 


0.93 


4.05 


3.55 


0.90 


FAL83 


23 


172 




MN3105 


2 


26 


367 


891203X4765 


3.90 


3.77 


0.89 


3.80 


3.23 


0.96 


FAL83 


2^ 


173 




MN3105 


3 


30 


367 


643280X9157 


4.56 


4.80 


0.40 


4.88 


4.33 


0.96 


FAL83 


25 


174 




MN3105 


4 


23 


817 


3284x609175 


4.64 


4.74 


0.53 


4.86 


4.30 


0.78 


FAL83 


26 


175 




MN3140 


1 


30 


815 


04291387X65 


4.24 


4.13 


0.81 


4.19 


3.57 


0.96 


FAL33 


27 


176 




MN3140 


2 


21 


847 


034X2198765 


4.49 


4.43 


0.73 


4.63 


4.00 


0.95 


FAL83 


28 


177 




MN3140 


3 


17 


837 


02419X36875 


4.35 


4.24 


0.73 


4.29 


3.71 


0.94 


FAL83 


29 


178 




MN3161 


1 


28 


817 


0849321X675 


4.67 


4.82 


0.38 


4.89 


4.29 


0.93 


FAL83 


30 


179 




MN3161 


2 


27 


847 


0493X268175 


3.48 


3.15 


1 .20 


3 . 17 


2.74 


0.96 


FAL83 


31 


180 




MN3161 


3 


22 


837 


40329X86175 


3.62 


3.32 


1 .02 


3.17 


2.91 


0.95 


FAL83 


32 


181 




MN3301 


0 


7 


837 


29108X74356 


3.76 


3.29 


0.70 


3.33 


3.43 


0.17 


FAL83 


33 


132 




MN3302 


0 


23 


815 


234109X8756 


4.37 


4.67 


0.56 


4.80 


4.17 


0.91 


FAL83 


3^ 


183 




MN3303 


0 


10 


815 


843021X9765 


4.92 


5.00 


0 . 00 


5.00 


4.70 


0.90 


FAL83 


35 


184 




MN3371 


0 


11 


367 


1874X309625 


4.45 


4.27 


0.36 


4.58 


4.27 


0 . 09 


FAL83 


56 


185 




MN3373 


0 


19 


813 


2198430X765 


4.16 


4.11 


0.85 


4.20 


3.47 


0.56 


FAL83 


37 


186 




MN3375 


0 


16 


827 


243091X8765 


4.58 


4.56 


0.70 


4.77 


4.19 


0.00 


FAL83 


38 


187 




MN4105 


1 


20 


837 


42X08931657 


4.27 


4.15 


0.91 


4.33 


3.90 


0 . 95 


FAL83 


39 


133 




MN4105 


2 


15 


837 


829041X7365 


3.81 


3.93 


1 . 18 


4.33 


3.20 


0.93 


FAL83 


^0 


189 




MN4105 


3 


22 


817 


0834921X576 


4.28 


4.18 


0.72 


4.20 


3.95 


0.91 


FAL83 


^1 


190 




MN4105 


4 


18 


367 


8432109X657 


4.33 


4.44 


0.60 


4.50 


3.78 


0.50 


FAL83 


^2 


191 




MM4110 


0 


9 


847 


72198403X56 


2.55 


1.89 


0.87 


1.75 


1 .89 


0.89 


FAL83 


43 


192 




MN4116 


0 


11 


857 


123497085X6 


4.37 


4 . 09 


0.79 


4.13 


4 .00 


0.64 


FAL83 


44 


193 




MN4122 


0 


11 


837 


4810923X765 


3.44 


3.18 


1 . 03 


3.20 


2.64 


0.10 


FAL83 


45 


194 




MN4123 


0 


10 


857 


2941X830657 


4.72 


4.60 


0.66 


4.79 


4.40 


0.90 


FAL83 


46 


195 




MN4145 


1 


26 


837 


4813290X756 


3.86 


3.73 


1 . 06 


3.90 


2.88 


0 . 92 


FAL83 


47 


196 




:*1N4145 


2 


13 


837 


4903281X765 


4.16 


4.23 


0.70 


4.25 


3.23 


0.77 


FAL83 


48 


197 




MN4154 


1 


22 


837 


042X9831756 


4.00 


3.95 


0.64 


3.96 


3.32 


0,70 


FAL83 



EXHIBIT 2 



STATISTICAL DESCRIPTION OF QUANTITATIVE VARIABLES 



VARIABLE 


N 


MEAN 


STANDARD 

DEVIATION 


MINIMUM 

VALUE 


MAXIMUM 

VALUE 


STD ERROR 
OF MEAN 


SIZE 


1610 


20.607453<;2 


8.75684623 


2.00000000 


86 .00000000 


0.21824022 


MEANll 


1610 


A. 12931677 


0.54590950 


1 .37000000 


5.00000000 


0.01360529 


MEAN12 


1610 


A. 02601242 


0.71591653 


1 .08000000 


5.00000000 


0.01784224 


MEDIAN12 


1610 


4.10199379 


0.76191130 


1 . 04000000 


5 . 00000000 


0.01898854 


MEAfU3 


1610 


3.66124224 


0.68517204 


1 . 08000000 


5.00000000 


0.01707602 


REQDPR 


1610 


0.77300000 


0.26391706 


0 .00000000 


1 . 00000000 


0 .00657740 



EXHIBIT 



3 



PEARSON CORRELATION COEFFICIENTS / PROB > R UNDER H0-.RHO = 0 / N = 1610 





SIZE 


SIZE 


1.00000 
0 .0000 


MEANll 


-0 . 18710 
0.0001 


MEAN12 


“0.16391 
0 .0001 


KEDIAN12 


-0 . 1A323 
0.0001 


MEAN13 


-0.22566 
0 .0001 


REQDPR 


0.37251 
0 .0001 


YEAR 


0 . 08116 
0.0011 



MEAtai 


MEA^a2 


-0 . 18710 
0 . 0001 


-0.16391 
0 . 0001 


1 .00000 
0 . 0000 


0 .9668A 
0 . 0001 


0 .9668A 
0 .0001 


1 .00000 
0 .0000 


0.95310 

0.0001 


0 .987A9 
0.0001 


0 . V2A50 
0.0001 


0 .88870 
0 .0001 


-0.25961 

0.0001 


-0.23521 
0 . 0001 


0 . 02AAA 
0 . 3271 


0 . 0116A 
0 .6A07 



MEDIAta2 


MEAN13 


-0.1A323 
0 . 0001 


-0.22566 

0.0001 


0.95310 

0.0001 


0 .92A50 
0 . 0001 


0.987A9 
0 .0001 


0 .88370 
0 . 0001 


1 .00000 
0 .0000 


0.87351 
0 . 0001 


0.87351 

0.0001 


1 .00000 
0 . 0000 


-0.21968 
0 .0001 


-0 . 3093A 
0 . 0001 


0 . 0121A 
0 .626A 


0 . 0A379 
0 . 0790 



REQDPR 


YEAR 


0.37251 
0 . 0001 


0 . 08116 
0 . 0011 


-0.25961 
0 . 0001 


0 .02AAA 
0.3271 


-0.23521 
0 . 0001 


0 .0116A 
0 .6A07 


-0.21968 

0.0001 


0.0121A 
0 .626A 


-0.3093A 
0 .0001 


0.OA379 
0 . 0790 


1 .00000 
0 . 0000 


0.13151 
0 .0001 


0.13151 
0 . 0001 


1 .00000 
0 . 0000 



EXHXBXT 4 



FREQUENCY 

I 

I 

I 

220 + 
I 
I 
I 

200 + 
I 
I 
I 

180 + 
I 
I 
I 

160 + 
I 
I 

! 

1^0 + 
I 
I 
I 

120 + 
I 
I 
I 

100 + 
I 
I 
I 

80 + 
I 
I 
I 

60 + 
I 
I 
I 

<40 + 
I 
I 
I 

20 + 



FREQUENCY DISTRIBUTION FOR MEAN ITEM 12 (N = 1,610) 
FREQUENCY BAR CHART 



X* XX 
1 1 
2 4 



XX 

XX 

XX 

XX 

XX 

XX 

XX 

XX XX 

XX XX 

XX XX XX 



























XX 


XX 


XX 


XX 
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6 
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4 
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2 
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6 


8 


6 



MEAN12 MIDPOINT 



EXHIBIT 5 



REGRESSION OF MEAN ITEM 12 ON ALL QUANTITATIVE VARIABLES 



DEP VARIABLE: MEAN12 



ANALYSIS OF VARIANCE 



SUM OF 

SOURCE DF SQUARES 



MEAN 

SQUARE F VALUE PROB>F 



MODEL ^ 
ERROR 1605 
C TOTAL 1609 



65<^. 31180 163.57795 15^1.110 

170.359A0 0.1061^293 

82^.67120 



0001 



ROOT MSE 
DEP MEAN 
C. V. 



0.3257958 
A . 026012 
8 . 092271 



R-SQUARE 
ADJ R-SQ 



0 .793<4 
0.7929 



PARAMETER ESTIMATES 



PARAMETER 

VARIABLE DF ESTIMATE 



STANDARD T FOR HO: 

ERROR PARAMETER=0 PROB 



INTERCEP 1 
MEAN13 1 
SIZE 1 
YEAR 1 
REQDPCNT 1 



0 . A6760587 
0 .95000607 
0.002398057 
-0 . 011^07AA 
0.001080501 



0.0621805^ 

0.01262108 

0.001008A62 

0.003666325 

0.0003A53A0 



7.520 


0 


75.271 


0 


2.378 


0 


-3.111 


0 


3.129 


0 



III 

0001 

0001 

0175 

0019 

0018 



EXHIBIT 



6 






:r^ 



ANOVA OF MEAN ITEM 12 BY COURSE AND FACULTY 
GENERAL LINEAR MODELS PROCEDURE 



DEPENDENT 


VARIABLE: 


MEAN12 










SOURCE 




DF 


SUM OF SQUARES 




MEAN SQUARE 


F VALUE 


MODEL 




278 


5A1 .66375A89 




1 .9A8A3077 


9.16 


ERROR 




1331 


283.007AAA86 




0.21262768 


PR > F 


CORRECTED 


TOTAL 


1609 


82A. 67119975 






0.0 


R-SQUARE 




C.V. 


ROOT MSE 




MEAN12 MEAN 




0 .65682A 




11 . A53A 


0.A6111569 




A.026012A2 




SOURCE 




DF 


TYPE I SS 


F 


VALUE PR > F 




COURSE 

FACULTY 




130 

1A8 


287 .9AA1695A 
253.71958535 




10. A2 0.0 

8.06 0.0 




SOURCE 




DF 


TYPE III SS 


F 


VALUE PR > F 




COURSE 

FACULTY 




128 

l»-*8 


62.96313651 

253.71958535 




2.31 0.0001 

8.06 0.0 





EXHIBIT 



V 



C\J 



Ill o 

h- ^ 

— UL 
Z -O 

LU c\3 

CD 
CO 



O 

lZ o 

< T 

< -a 




CO 



E2CHXBIT 



ANCOVA OF MEAN ITEM 12 BY COURSE AND FACULTY 
WITH MEAN ITEM 13 AS A COVARIATE 

GENERAL LINEAR MODELS PROCEDURE 



DEPENDENT 


VARIABLE: 


MEAN12 








SOURCE 




DF 


SUM OF SQUARES 


MEAN SQUARE 


F VALUE 


MODEL 




279 


738.395087A0 


2.64657737 


o 

00 

o 


ERROR 




1330 


86.27611235 


0.06486926 


PR > F 


CORRECTED 


TOTAL 


1609 


824.67119975 




0.0 


R-SQUARE 




C.V. 


ROOT MSE 


MEAN12 MEAN 




0.895381 




6.3262 


0.25469444 


4.02601242 




SOURCE 




DF 


TYPE I SS 


F VALUE PR > F 




MEAN13 




1 


651 .31623693 


10040.45 0.0 




COURSE 




130 


46 .44339437 


5.51 0.0001 




FACULTY 




1A8 


40.63545610 


4.23 0.0001 




SOURCE 




DF 


TYPE III SS 


F VALUE PR > F 




MEAfll3 




1 


196.73133251 


3032.74 0.0 




COURSE 




128 


18.89818300 


2.28 0.0001 




FACULTY 




1A8 


40.63545610 


4.23 0.0001 





EXHXBIT 



9 



CO 

E 

CD 



CNJ 



03 

Q> 



LU 

h- 



< 

LU 



— LL 
Z -D 



= 0) 

§1 

03 ^ 



§ 

cc 



03 

CO 

03 



Q> 

CO 

L- 

D 

O 

O 



CD 

D 

■o 




(0 

o 
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liJ 

15 

c 
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E 

0) 

k_ 

o 

c 



CO 
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• 4 — 
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0) 
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0) 
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APPENDIX A 



FORTRAN Program 



FILEi SELECTS FORTRAN A1 



//SELECTS JOB (0597,0828), 'SELECTS FORTRAN CL ASS=A 
//XMAIN SYSTEM=SY2 
// EXEC VSF2CG 
//FORT.SYSIN DD X 

CHARACTER x 5 TRM 

DIMENSION XITEM(16),YITEM(16),ITEM(16),MCURR(10),KCURR(10),XN(16) 
DIMENSION P(5) 

TRM = 'HTR83* 

MSOF = 0 
N = 0 
YIl = 0. 

YY12 = 0. 

REQ = 0. 

PREQ = 0. 

XREQ = 0. 

DO 3 I = 1, 5 
P(I) = 0. 

3 CONTINUE 

DO 5 I = 1, 16 
YITEM(I) = 0. 

XN(I) = 0. 

5 CONTINUE 

DO 6 I = 1, 10 
KCURR(I) = 0 
MCURR(I) = 0 

6 CONTINUE 

10 READ(5,100,END=300) NSOF, NCURR, REQ, ( XITEMC I ) , I =1 , 16 ) 

100 FORMAT(3X,I3,I3,<4X,F1.0,16(F1 .0)) 

9 CONTINUE 

IF(NSOF.LT.IAO) GO TO 10 
IF(NSOF.EQ.MSOF) GO TO 20 
IF(MSOF.EQ.O) GO TO 19 
XMED = 0. 

IF(XN(12).EQ.O. ) GO TO A 
XN12 = XN(12) 

YY12 = YY12/XN12 
XN2 = XN12/2. 

SUMP = 0. 

DO 2 I = 1, 5 
IF(SUMP.GE.XN2) GO TO 2 
III = I 

SUMP = PCI) + SUMP 
2 CONTINUE 

SUMP = SUMP - P(III) 

XI = FLOAT(III) 

XI2 = XI - .5 

XMED = XI2 + (XN2 - SUMP)/P(III) 

^ CONTINUE 

DO 11 I = 1, 16 
IF(XN(I) .EQ.O. ) GO TO 11 
YITEM(I) = YITEM(I)/XN(I) 

XN(I) = 0. 

11 CONTINUE 

Y12 = YITEMC12) 

Y13 = YITEMC13) 

YY12 = SQRTCYY12 - Y12XY12) 

DO 8 K = 1, 11 
Yll = YITEM(K) + Yll 
YK = 5. 

DO 7 I = 1, 11 
IFCXNCI) .EQ.l . ) GO TO 7 
IF(YITEM(I).GT.YK) GO TO 7 
YK = YITEM(I) 

II = I 

7 CONTINUE 
XN(II) = 1. 

IF(II.EQ.IO) II = 0 
ITEM(K) = II 

8 CONTINUE 
MAX = 0 

DO 12 I = 1, 10 
IFCKCURRd) .LE.MAX) GO TO 12 



FILEi SELECTS FORTRAN A1 



MAX = KCURR(I) 

NCURR = MCURR(I) 

12 CONTINUE 

R = PREQ/XREQ 
XH = XMED 
Y = Yll/11 . 

HRITEC 1,200) MS0F,N,NCURR,(ITEM(I),I=1,11),Y,Y12,YY12,XM,Y13,R,TRM 
200 F0RMAT(1X,I3,3X,I2,3X, 13, 3X, 1 1 ( I 1) , 3X, 6 ( FA . 2, 3X) , A5 ) 

N = 0 
YU = 0. 

YY12 = 0. 

REQ = 0. 

PREQ = 0. 

XREQ = 0, 

DO 15 I = 1, 16 
YITEM(I) = 0, 

XN(I) = 0. 

15 CONTINUE 
MSOF = NSOF 

DO 16 I = 1, 10 
KCURR(I) = 0 
MCURR(I) = 0 

16 CONTINUE 

DO 17 I = 1, 5 
PCI) = 0. 

17 CONTINUE 

IFCN50F.GT.219) GO TO 300 
GO TO 9 

19 MSOF = NSOF 

20 CONTINUE 
KK= 0 

DO 25 I = I, 10 
IF(KK.EQ.l) GO TO 25 
IF(MCURRd) .NE. NCURR) GO TO 21 
KCURR(I) = KCURR(I) + 1 
KK= 1 
GO TO 25 

21 IF(MCURRd) .NE.O) GO TO 25 
MCURRCI) = NCURR 
KCURRCI) = KCURR(I) + 1 

KK - 1 

25 CONTINUE 
N = N + 1 

X12 = XITEMC12) 

IFCX12.LT.1. .0R.X12.GT.5.) GO TO 26 
N12 = IFIXCX12) 

P(N12) = PCN12) + 1. 

26 CONTINUE 

IFCREQ. NE.O. .AND. REQ. NE.l.) GO TO 27 
PREQ = REQ + PREQ 
XREQ = XREQ + 1. 

27 CONTINUE 

DO 30 I = 1, 16 

IF(XITEM(I).LT.l. .0R.XITEMd).GT.5.) GO TO 30 
YITEMCI) = XITEMCI) + YITEMCI) 

XNCI) = XNCI) + 1. 

IFd.EQ.12) YY12 = X12XX12 + YY12 
30 CONTINUE 
GO TO 10 
300 CONTINUE 
STOP 
END 
/X 

//GO.FT05F001 DD DISP=SHR, DSN=MSS . F0597 . R0N(MNT83 ) 

//GO.FTOlFOOl DD SYSOUT=B, DCB=BLKSIZE=60 
// 



APPENDIX B 



SAS Program 



FILE 


1 MERGE2 


SAS 


B1 


CMS 


FILEDEF SCORES DISK 


FAL90 SOFDATA A; 


CMS 


FILEDEF NAMES DISK 


FAL90 SOFNAME B; 


CMS 


FILEDEF MERGE DISK 


FAL90 SOFMERGE B; 


DATA 


SOFDATA; 








INFILE SCORES; 






INPUT 32 


IDSOF 


$CHAR3. 




37 


NSTUDENT 


SCHAR5. 




313 


CURR 


$CHAR3. 




319 


RANKS 


$CHAR11 . 




333 


MEANll 


$CHAR<i . 




3A0 


MEAN12 


$CHARA, 




3A7 


STD12 


$CHARA. 




35A 


MED12 


$CHARA. 




361 


MEAN13 


$CHARA. 




368 


REQD 


$CHARA. 




375 


QTRYR 


$CHAR5. ; 


DATA 


SOFNAME; 








INFILE NAMES; 






INPUT 32 


IDSOF 


$CHAR3. 




38 


IDPROF 


$CHAR2. 




313 


IDCOURSE 


$CHAR6. 




322 


SEGMENT 


$CHAR1. ; 


PROC 


SORT DATA 


=SOFDATA; 


BY IDSOF; 


PROC 


SORT DATA 


=SOFNAME; 


BY IDSOF; 


DATA 


MERGE; 







MERGE SOFNAME SOFDATA; BY IDSOF; 
OPTIONS LINESIZE = 80; 

PROC PRINT; 

DATA _NULL_; 

SET MERGE; 

FILE MERGE; 





IDSOF 


$CHAR3. 


+ 1 


IDPROF 


$CHAR2. 


+ 1 


IDCOURSE 


$CHAR6 . 


+ 1 


SEGMENT 


$CHAR1 . 


+ 1 


NSTUDENT 


$CHAR3. 


+ 1 


CURR 


$CHAR3, 


+ 1 


RANKS 


$CHAR11 . 


+ 1 


MEANll 


$CHARA. 


+ 1 


MEAN12 


$CHARA. 


+ 1 


STD12 


$CHARA . 


+ 1 


MED12 


$CHARA, 


+ 1 


MEAN13 


$CHARA. 


+ 1 


REQD 


$CHARA. 


+ 1 


QTRYR 


$CHAR5. ; 



APPENDIX C 



Average Mean Item 12 Responses for Courses 



a 



MEAN ITEM 12 FOR EACH COURSE 
GENERAL LINEAR MODELS PROCEDURE 



COURSE 

AS1501 

AS1701 

AS3501 

AS3610 

AS3611 

ASA601 

ASA610 

ASA613 

CM3001 

CM3002 

CM3111 

CM3112 

CM3212 

CMA003 

CMA502 

CMA925 

C03111 

IS0123 

ISIOOA 

IS2000 

IS2100 

IS2901 

IS3000 

IS3020 

IS3100 

IS3170 

IS3171 

IS3133 

IS318A 

IS3186 

IS3220 

IS3502 

IS3503 

ISA182 

ISA183 

ISA18A 

IS^185 

ISA200 

ISA300 

ISA320 

ISA502 

ISA925 

MG3373 

MM3301 

MfUlll 

MN1501 

MN2013 

MN2031 

MN2105 

MN2106 

MN2111 

MN2112 

MN2113 

MN211A 

MN2115 

MN2150 

MN2155 

MN2302 

MN2901 

MN3001 

MN3031 

MN3101 

MN310^ 

MN3105 

MN3111 

MN311A 

MN3116 

MN3123 

MN31A0 

MN3161 

MN3172 



MEAN12 

A. A2588235 
A. A1500000 
A. 03000000 
3.63222222 
A.51A66667 
A. 09500000 

3.00000000 
3.75000000 
A . A0333333 
3.90A00000 
3.62777778 
3.13750000 
3.07000000 
A. 25000000 
A . A1333333 
3.9228571A 
2.61500000 
A.6A529A12 
3 .15000000 
3.18A09091 
2.915789A7 
A. 78937500 
3.88230769 
3.27000000 
3.7A000000 
3.9A760000 
3.78692308 
3.91133333 
A. 18000000 

5.00000000 
A. 17100000 
3 .6929A118 
3.82000000 
A.3213636A 
A . 06A50000 
3.9A000000 
3.89392857 
3 .69AA0000 
3.92500000 
A. AOOOOOOO 
A .32000000 
A.A9000000 

2.00000000 
A. 17000000 
3 .5A000000 
A .71000000 
A. 53000000 
3 .31A75000 
A.6A500000 
A. 17625000 
A. 28666667 
A. 10000000 
A. 39000000 
3.51500000 

78000000 
A. 35170213 
A. 32111111 
A. A0555556 
^.53000000 
3 . 18250000 
2.72000000 
A. 36200000 
3.03000000 
A.1AA79A52 
3.90187500 
A. 76600000 
4.19000000 
4.72750000 
3.77977778 
4.17217391 
3.54204545 



MEANS 

N 

17 

2 

10 

18 

15 

2 

2 

1 

6 

5 

18 

4 

1 

1 

3 

7 

2 

17 

4 

22 

19 

16 

13 

2 

3 

25 

13 

45 

1 

2 

10 

17 

3 

22 

20 

6 

28 

25 

10 

1 

1 

6 

1 

1 

1 

3 

1 

40 

2 

8 

12 

9 

8 

2 

1 

47 

27 

18 

15 

4 

1 

5 

1 

73 

16 

5 

2 

4 

45 

46 

44 



MEAN ITEM 12 FOR EACH COURSE 



GENERAL LINEAR MODELS PROCEDURE 
MEANS 



COURSE 


N 


MEAN12 


MN3301 


44 


4.09977273 


MN3302 


7 


3.78571429 


MN3303 


12 


4.49833333 


MN330A 


13 


4.48923077 


MN3305 


12 


4.26666667 


MN3307 


22 


3.91954545 


MN3310 


2 


4.16500000 


MN3333 


58 


3.92103448 


MN3371 


18 


4.48555556 


MN3372 


13 


3.30076923 


MN3373 


7 


4.09142857 


MN337A 


16 


3.87625000 


MN3375 


5 


4.59800000 


MN3377 


11 


3.68363636 


MN3650 


1 


5.00000000 


MN3760 


14 


4.17500000 


MN3801 


6 


4.62333333 


MN3900 


1 


5 .00000000 


MN3902 


11 


3.37181818 


MN3903 


4 


2.83750000 


MNA105 


54 


4.38962963 


MNA106 


16 


4.34562500 


MNAllO 


15 


3.65533333 


MNAlll 


5 


3.63000000 


MNA112 


3 


4.2e666667 


MNA116 


4 


3.72000000 


MNA117 


3 


4.20666667 


MNA119 


4 


4.40750000 


MNA120 


1 


4.65000000 


MNA121 


4 


4.64250000 


MN4122 


9 


^.^2444444 


MNA123 


4 


4.24000000 


MNA12A 


4 


4.57250000 


MNA125 


9 


4 . 42444444 


MNA127 


4 


4.65250000 


MNA145 


40 


3.15875000 


MNA151 


18 


4.10555556 


MNA152 


16 


4.19625000 


MNA15A 


68 


3.91176471 


MNA155 


12 


4.19166667 


MNA159 


12 


4.69750000 


MNA161 


23 


4.10130435 


MNA162 


17 


3.96294118 


MNA163 


16 


4.72500000 


MNA301 


11 


4 .68818182 


MNA302 


5 


4.70000000 


MNA310 


10 


3.92600000 


MN4371 


12 


4 . 40750000 


MNA372 


4 


4.65750000 


MNA373 


8 


4.73000000 


MN437A 


1 


4.92000000 


MN4500 


10 


3.98300000 


MN4761 


14 


4.12214286 


MN4900 


2 


4.62500000 


MN4904 


1 


5 .00000000 


MN4942 


1 


3.50000000 


MN4960 


1 


4.67000000 


MN4970 


2 


4.80000000 



A gency 
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